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Abstract 

Background: VJe previously developed and validated the Oxford NOTECHS rating system for evaluating the non-technical 
skills of an entire operating theatre team. Experience with the scale identified the need for greater discrimination between 
levels of performance within the normal range. We report here the development of a modified scale (Oxford NOTECHS II) to 
facilitate this. The new measure uses an eight-point instead of a four point scale to measure each dimension of non- 
technical skills, and begins with a default rating of 6 for each element. We evaluated this new scale in 297 operations at five 
NHS sites in four surgical specialities. Measures of theatre process reliability (glitch count) and compliance with the WHO 
surgical safety checklist were scored contemporaneously, and relationships with NOTECHS II scores explored. 

Results: Mean team Oxford NOTECHS II scores was 73.39 (range 37-92). The means for surgical, anaesthetic and nursing 
sub-teams were 24.61 (IQR 23, 27); 24.22 (IQR 23, 26) and 24.55 (IQR 23, 26). Oxford NOTECHS II showed good inter-rater 
reliability between human factors and clinical observers in each of the four domains. Teams with high WHO compliance had 
higher mean Oxford NOTECHS II scores (74.5) than those with low compliance (71.1) {p = 0.010). We observed only a weak 
correlation between Oxford NOTECHS II scores and glitch count; r=-0.26 (95% CI -0.36 to -0.15). Oxford NOTECHS II 
scores did not vary significantly between 5 different hospital sites, but a significant difference was seen between specialities 
(p = 0.001). 

Conclusions: Oxford NOTECHS II provides good discrimination between teams while retaining reliability and correlation 
with other measures of teamwork performance, and is not confounded by technical performance. It is therefore suitable for 
combined use with a technical performance scale to provide a global description of operating theatre team performance. 
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Introduction 

The evaluation of team non-technical skills has become 
important in research on surgical safety because of the evidence 
that teamwork glitches, communication failures, cultural and 
hierarchal barriers contribute to safety failures [1-4]. Comparisons 
between the work environments of airline cockpits and operating 
theatres encouraged researchers to apply techniques originally 
developed to evaluate team interaction within the aviation 
industry to situations in healthcare [5-7] . 

Non-technical skills assessment tools adapted from aviation have 
been used in the operating theatre to understand the influence of 
behaviour on outcome. A number of methods have been 
developed for assessing teamwork skills in operating theatres, 
based on direct observation or video analysis[l,8-14]. The 
development of the first NOTECHS assessment method was 
conducted via a European collaboration [15] which has since been 



followed by further developments. OTAS[16]; Oxford NO- 
TECHS[14], OSTAS[17] and EPOC[18] provide whole team 
assessments; ANTS [10], NOTSS[19] and SPLINTS [20] focus on 
sub-team performance. The methodology underpinning the 
Oxford NOTECHS tool maps directly to the original methods 
used in the aviation industry via a carefully planned adaptation for 
use with surgical teams[14,21]. It has since been successfully 
applied in a range of surgical settings examining the effects of 
Crew Resource Management (CRM)-based training courses on 
surgical teams[22,23]. 

As the complexity of surgical team working becomes clearer, so 
the demands for more sophisticated methods of measurement have 
followed. The validity and reliability of the Oxford NOTECHS 
system was demonstrated in live theatre environments [14], but a 
number of imperfections were noted in discussion and through 
experience. The system tended to group results close to the 
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median, and therefore had a suboptimal capacity to discriminate 
between teams whose performance was not extreme. 

As part of a wider project aimed at extensively examining the 
effects of a theatre based teamwork training intervention, we took 
the opportunity to restructure the original Oxford NOTECHS 
tool (Oxford NOTECHS II). The purpose of this re-design was to 
develop a tool with improved scalabiUty, enabling a richer 
understanding on the impact of quahty improvement interven- 
tions. This study describes and evaluates the new scale (Oxford 
NOTECHS II). 

Methods 

Ethics Statement 

Ethics Committee approval was obtained for this study (NRES 
Committee South Central - Oxford A REC 09/H0604/39). 
Hospital management and all theatre staff were fully informed of 
the study and provided written consent to take part during the 
observation period. 

Oxford NOTECHS II is built on extensive work in developing, 
evaluating, and vaUdating the original Oxford NOTECHS [14]. 
The final version was arrived at through group discussion between 
members of the original (PM, KC) and new group members (ER, 
LM, MH and SP), statistical modelling (GC) followed by iterative 
small-scale testing. The four original Oxford NOTECHS domains 
of: leadership and management (L&M); teamwork and coopera- 
tion (T&C); problem-solving and decision-making (PS & DM); and 
situation awareness (SA) remain unchanged. The- bcha\ioural 
markers for each of the Oxford NOTECHS II parameters are 
largely unchanged from before (Table 1). There is no alteration to 
the consideration of the theatre team as three sub-teams: surgical 
(operating and assisting surgeons); anaesthetic (anaesthetists and 
anaesthetic nurses/practitioners) and nursing (scrub and non- 
anaesthetic circulating nurses and practitioners). 

The differences from the original Oxford NOTECHS are the 
use of an eight point instead of a four-point s[:ale for dimensions; 
assigning all teams a baseline score of 6, a behavioural marker of 
'consistently maintaining an effective level of patient safety and 
teamwork'(Table 2), from which subsequent observations could 
result in dc\iati()n upwards or downwards in response to 
observable behavioural markers [14]. 

Training of observers 

Observations of intra-operative process and non-technical skills 
were performed real-time in the operating theatre by two 
observers, one being a clinical and one a human factors (HE) 
specialist. The clinical observers were either surgical trainees (ER 
& MH) or operating department practitioners QM) with > 1 years 
theatre experience. The HE specialists had at least an undergrad- 
uate qualification in human factors (LM, SP and LB). The clinical 
observers gained experience of HF principles from in-house 
lectures and literature reviews and the HF observers gained 
experience of the theatre environment from theatre observational 
practice and mentoring by cKnical observers. All observers trained 
in the use of the Oxford NOTECHS II over a one month period 
with self-study and group practice sessions using video recordings 
of operating teams in simulated settings. 

Logistics of scoring Oxford NOTECHS II 

The three theatre sub-teams are each scored on four parameters 
(L&M, T&C, PS & DM and SA) resulting in a total of twelve 
scores. The final scores were calculated at the end of the operation 
(during suturing/ applying dressings) and entered into procedure- 
specific data collection books in an independent manner[24]. 



Once the operation was complete, each observer's scores were 
individually entered on a secure database. 

The data for this study was collected as part of a multi-centre 
observational study at five hospitals from three NHS trusts, the 
Safer Delivery of Surgical Services Programme (S3). The hospitals 
comprised: one district general hospital, three teaching hospitals 
and one tertiary referral centre. A variety of surgical specialities 
were selected for observation including: elective orthopaedic; 
trauma orthopaedics; plastic surgery and vascular/ general surgery. 
Within each surgical speciality a range of surgical interventions 
were selected for inclusion in the study based on frequency, and 
operative lists were targeted with the highest proportion of this 
pre-determined case mix (Table 3). This was therefore a large 
convenience sample with predefined characteristics. 

Validation 

Alongside non-technical skills rating, observers collected infor- 
mation on the technical performance of the team and the quality 
and completeness of the WHO time-out procedure. When the 
WHO time-out was attempted, observers critiqued process quality 
using three binary (yes/no) parameters: was all information 
communicated; were all team members present and was there 
active participation. Observers recorded these parameters inde- 
pendentiy, then discussed their findings at the conclusion of the 
case to reach agreement on a joint final score [25]. Intra-operative 
process disruptions (e.g. distractions, int(;rruptions, planning and 
preparation issues) were evaluated using the glitch method[26]. 
The associations between both the glitch count and WHO time- 
out performance and Oxford NOTECHS II scores were subjected 
to exploratory statistical analysis to provide independent validation 
of the new scale, in addition to that provided for the original 
Oxford NOTECHS[14]. 

To further explore the relationship between Oxford NO- 
TECHS II scores and glitch count, a Delphi experiment was 
undertaken within the research group. The researchers were asked 
to predict the relationship between Oxford NOTECHS II scores 
and the 14 glitch sub-categories[26]. This experiment was 
undertaken in two rounds, with researchers ranking the likelihood 
of correlation of each sub-category with Oxford NOTECHS II. At 
the end of the second round, four sub-categories were grouped as 
'likely to correlate', six as 'may or may not correlate' and four as 
'wUl not correlate' (Table 4). A separate analysis of the association 
between each of these sub-categories and Oxford NOTECHS II 
was carried out. 

Statistical methods 

Agreement between observers (clinical and HF) for each 
dimension of the Oxford NOTECHS II instrument was assessed 
using agreement and the intraclass correlation coefficient. Mean 
Oxford NOTECHS II scores were compared for operations with 
high and k)w WHO time-out scores[25] using the t-test. 
DifiFerences between mean Oxford NOTECHS II scores for 
different specialities and sites were examined using analysis of 
variance. Where differences between specialties or between sites 
were observed, the t-test was used to explore where the difference 
lay; no adjustment was made for multiple testing. The correlation 
between Oxford NOTECHS II scores and glitch rate was 
explored graphically and quantified by Pearson's correlation 
coefficient. We considered P values <0.05 to be statistically 
significant. All statistical analyses were carried out in R.3.0.1. 
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Table 1. Operating theatre team Non-Technical Skills (NOTECHS) assessment tool[14]. 




Leadership and management 


Leadership 


Involves/reflects on suggestions/visible/accessible/inspires/motivates/coaches 


Maintenance of standards 


Subscribes to standards/monitors compliance to standards/intervenes if deviation/deviates with team 
approval/demonstrates desire to achieve high standards 


Planning and preparation 


Team participation in planning/plan is shared/understanding confirmed/projects/changes in consultation 


Workload management 


Distributes tasks/monitors/reviews/tasks are prioritised/allots adequate time/responds to stress 


Authority and assertiveness 


Advocates position/values team input/takes control/persistent/appropriate assertiveness 


Teamwork and co-operation 


Team building/maintaining 


Relaxed/supportive/open/inclusive/polite/friendly/use of humour/does not compete 


Support of others 


Helps others/offers assistance/gives feedback 


Understanding team needs 


Listens to others/recognises ability of team/condition of others considered/gives personal feedback 


Conflict solving 


keeps calm in conflicts/suggests conflict solutions/concentrates on what is right 


Problem-solving and decision-making 


Definition and diagnosis 


Uses all resources/analytical decision making/reviews factors with team 


Option generation 


Suggests alternative options/asks for options/reviews outcomes/confirms options 


Risk assessment 


Estimates risks/considers risk in terms of team capabilities/estimates patient outcome 


Outcome review 


Reviews outcomes/reviews new options/objective, constructive and timely reviews/makes time for review/ 
seeks feedback from others/conducts post treatment review 


Situation awareness 


Notice 


Considers all team elements/asks for or shares information/aware of available of resources/encourages 
vigilance/checks and reports changes in team/requests reports/updates 


Understand 


Knows capabilities/cross checks above/shares mental models/speaks up when unsure/updates other team 
members/discusses team constraints 


Think ahead 


Identifies future problems/discusses contingencies/anticipates requirements 


doi:1 0.1 371 /journal.pone.0090320.t001 



Results 

A total of 297 operations were observed, lasting an 
average of 1 hour 53 minutes. Site A was the primary 
study site, and so had the largest number of operations 
(Table 5). 

An initial live test of inter-rater reliability in 20 elective 
orthopaedic operations across multiple sites showed good inter- 
rater agreement [2 7]. Subsequent analysis on the whole dataset 
confirmed the maintenance of good inter-rater agreement for total 
Oxford NOTECHS II score between the HF and clinical 
observers (Figure 1 and Table 6). 

On sub-category analysis we generally found good levels of 
agreement as evaluated by weighted intraclass correlation 



coefficient, but some sub-categories had low values despite the 
observed agreement being high. This was particularly seen in the 
Problem Solving and Decision Making category for nursing and 
anaesthetic teams and the Teamwork and Cooperation category 
for the nursing team. These findings are largely explained by the 
high concentration of ratings by the two observers (HF and 
clinical) around a single Oxford NOTECHS II score. 

Oxford NOTECHS II scores were significandy higher 
(p = 0.01) for teams who met all 3 quality criteria for the WHO 
time-out (mean Oxford NOTECHS II score 74.5 +/- SD = 7.3, 
n = 98) compared to those who met none of them (mean Oxford 
NOTECHS II score 71.1 +/- SD = 8, n = 54). There was only a 
weak correlation between NOTECHS II score and glitch rate 
(r=-0.26, 95% CI -0.36 to -0.15, Figure 2). We found no 



Table 2. Behavioural parameters of Oxford NOTECHS 





Behaviour 


Frequency 


Oxford NOTECHS II score 


Compromises patient safety and effective team work 


Consistently 


1 




Inconsistently 


2 


Could directly compromise patient safety and effective team work 


Consistently 


3 




Inconsistently 


4 


Maintains an effective level of patient safety and teamwork 


Inconsistently 


5 




Consistently 


6 


Enhances patient safety and effective teamwork 


Inconsistently 


7 


Consistently 8 



doi:l 0.1 371 /journal.pone.0090320.t002 
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Table 3, Surgical case mix. 




Surgical Speciality 


Types of operation 


Elective Orthopaedics 


Primary and revision knee and hip arthroplasty and arthroscopic procedures 


Trauma Orthopaedics 


Manipulation of fractures and dislocations under anaesthetic, open reduction and either internal fixations or hemi- 
arthroplasty procedures 


Vascular Surgery 


Arterial bypass, endarterectomies and hernia repair 


Plastic Surgery 


Excision of benign and malignant lesions with a range of closure techniques including free flaps and upper limb and nerve 
surgery 



doi:l 0.1 371 /journal.pone.0090320.t003 



associations between whole or sub-team Oxford NOTECHS II 
scores and sub-category glitch rates (Figure 2). 

The range and the distribution of Oxford NOTECHS II scores 
for, the sub-teams and the whole theatre team are displayed in 
Figure 3. There was no statistically significant difference in mean 
Oxford NOTECHS II scores between the three theatre sub-teams 
(surgical team 24.61 (IQR 23, 27); anaesthetic team 24.22 (IQR 
23, 26) and nursing team 24.55 (IQR 23, 26)) p = 0.235. The 
surgical team appeared to vary most and the anaesthetic team the 
least across all four parameters. 

In elective orthopaedic surgery the difference in mean Oxford 
NOTECHS II at four hospital sites (from 72.85 to 75.23) was not 
significant (p = 0.643). In vascular surgery, the mean Oxford 
NOTECHS II score at two hospital sites (from 69.75 to 73.24) was 
not significant (p = 0.089). Across chnical specialties there was 
statistically significant heterogeneity in Oxford NOTECHS II 
score, which ranged from 67.14 in plastic surgery to 73.86 in 
elective orthopaedic surgery (p = 0.001). This was explained by 
significant differences between elective orthopaedic and plastic 
surgery (means 73.86) vs 67.14, difference 6.72, 95% CI 1.48 to 
11.97), and between trauma orthopaedics and plastics (means 73 
vs 67.14, difference 5.86, 95% CI 0.01 to 11.63). 



Table 4. Results of the team Delphi experiment, categorising 
the predicted relationship between Oxford NOTECHS II and 
glitch category. 



Delphi results 



Glitch categories with Delphi ranking 



Likely to correlate 



Potentially correlate 



Will not correlate 



1. Planning and preparation 

2. Communication 

3. Absence 

4. Process deviation 

5. Training 

6. Distractions 

7. Health and safety 

8. Workspace 

9. IVlaintenance 

10. Maintenance equipment 
Slips 

Environment 
Equipment design 
Patient related factors 
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Discussion 

We set out to develop an improved non-technical skills 
assessment scale which would provide greater discrimination 
between the behaviour of theatre teams, as we felt that the 
tendency of the previous Oxford NOTECHS scale to score most 
teams within a fairly narrow middle range may have limited its 
value in studies of the effects of interventions. This was the 
rationale for doubling the scale and setting a default value close to 
the equivalent of the median result found in previous studies. 
Anchoring teams on a starting score of 6 was intended to maintain 
consistency and frame the observations in the light of positive 
assumptions about motivation and professionalism [28] . 

The new scale shows good inter-rater reliability between 
observers from different backgrounds across five hospital sites 
and four surgical specialities. This suggests that it may be possible 
for one observer from either a clinical or HF background to 
successfully rate non-technical skills using Oxford NOTECHS II 
after limited training, which would have important resource 
implications. However further work will be required to demon- 
strate this. 

The final version of the score had improved face validity 
compared to the original Oxford NOTECHS amongst the 
research team. We did not alter the fundamentals of the scale, 
and therefore believe that the validation work performed for 
Oxford NOTECHS [14] remains relevant. However we studied 
association between Oxford NOTECHS II and both WHO time- 
out performance and glitch count, to assess concurrent and 
construct validity for the new scale. We found a statistically 
significant association between high quality WHO time-out and 
higher Oxford NOTECHS II scores. This finding adds support to 
the hypothesised mechanistic effect of a well performed WHO 
surgical safety checklist on intra-operative non-technical skills 
performance. This may be of particular import when teams 
frequently change - a characteristic of aviation and medical teams. 

In contrast to previous work [14], we only found a weak 
correlation between Oxford NOTECHS II and glitch rate. We 
studied this association in more detail by developing a consensus 
grading, from strong to none, for the associations with the new 
scale we expected to see for different types of glitch, based on our 
subjective impression of their relationship with the quality of 
teamwork. Subsequent analysis however showed no association for 
any of the sub-categories. Technically this may have been because 
the glitch score represents a substantial re-categorisation of intra- 
operative events compared to the NOPE (non-operative proce- 
dural errors) system used in previous work[29]. Conceptually, it is 
superficially surprising that this is the case: one could imagine a 
poorly performing surgical team doing 'badly' by having a poor 
non-technical skills and this being connected to a large number of 
glitches. However, many glitches are caused by a range of factors 
exogenous to the team (e.g. interruptions or poor equipment 
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Table 5. Number of operations performed by site and 
speciality. 





Number of 












Total 


operations 


Site A 


Site B 


Site C 


Site D 


Site E 


specialty 


Elective Orthopaedics 


n = 101 


n = 26 


n = 54 


n = 30 




n 


= 211 


Trauma Orthopaedics 






n = 16 






n 


= 16 


Vascular surgery 




n = 21 






n = 27 


n 


= 48 


Plastic surgery 


n = 22 










n 


= 22 


Total per site 


n = 123 


n = 47 


n = 70 


n = 30 


n = 27 


n 


= 297 
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maintenance), whilst the occurrence of glitches may in some cases 
give teams the opportunity to exercise non-technical skills and 
achieve Oxford NOTECHS II scores of 7 or 8. This resonates 



with findings from previous research in the field recognising that 
resilience and the ability for teams to recover is an important 
factor in achieving safe care delivery[30,31]. The relationship 
between technical and non-technical performance thus appears to 
be more complex than we had previously anticipated. 

We consider that the glitch method measures the effect of the 
wider system upon the theatre process and the Oxford 
NOTECHS II measures the human contribution. It is generally 
agreed that a full assessment of theatre team performance requires 
evaluation of both technical and non-technical perfor- 
mance[14,21,32,33]. Our findings suggest that Oxford NO- 
TECHS II and glitch count are orthogonal in terms of what they 
measure, and therefore complement each other well in achieving 
this. 

An encouraging finding is that the majority of operations were 
performed by well-coordinated and functioning teams. With 
regards to differences between the theatre sub-teams, we found 
that the surgical team varied the most and utilised more of the 




40 60 80 100 

Total Oxford NOTECHS II score (clinical observer) 



Figure 1. Inter-rater agreement between HF and clinical observers. 

doi:1 0.1 371/journal.pone.0090320.g001 
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Table 6. Inter-rater agreement (% agreement, intraclass correlation coefficient). 







Surgical team 


Nursing team 


Anaesthetic team 


Leadership and management 


59%, 0.881 


59%, 0.777 


64%, 0.739 


Teamwork and cooperation 


55%, 0.757 


45%, 0.343 


64%, 0.676 


Problems solving and decision mal<ing 


55%, 0.725 


63%, 0.397 


78%, 0.385 


Situational awareness 


53%, 0.770 


48%, 0.683 


56%, 0.675 



doi:1 0.1 371 /journal.pone.0090320.t006 



scale than the anaesthetic or nursing team. We consider tliat this is 
due to the cultural and functional difTerences between the teams. 
The surgical team tends to comprise team members who are 
classical external processors, and their behaviour is necessarily 
linked to clearly observable technical tasks. The nursing team 
behaviour scores varied less, which may be due to within-team 
diversity. The nature of their work means that one team member 
(the scrub nurse) is entirely dependent upon the wider nursing 
team to support them, and where the behaviour of individuals 
within the team is very different, the team scores can become 
equivocal. The anaesthetic team's scores varied the least of all. 
Their work often requires little communication with the wider 
team although anaesthetists and anaesthetic support staff are in 
frequent communication. They also tend to make and action 
decisions from internally derived situational considerations. This 
makes the rating of anaesthetic behaviour dilTicult and this was 
borne out in our Oxford NOTECHS II sub-category analysis. 
This finding reproduced those of our original NOTECHS 
study [14]. As before, we have not weighted the significance of 
one team's contribution to the overall score as it remains unknown 
what significance each team's contribution has on the overall 
safety of the operating theatre. Further investigation may reveal 
that some teams contribute more to overall theatre team safety 
than others. 
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Figure 2. Relationship between glitch rate (total glitch rate per 
operation per hour] and total Oxford NOTECHS II score. 
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Due to the study design we were able to compare mean Oxford 
NOTECHS II scores across difFerent sites and surgical specialities. 
There was no dilTerence between the four NHS sites where 
elective orthopaedics or the two sites where vascular/general 
surgery was performed. We did however find differences between 
surgical specialities, with elective orthopaedic teams demonstrating 
higher Oxford NOTECHS II scores. This finding may demon- 
strate differences in the team dynamics due to the nature of the 
operation - with some requiring more overt team coordination 
than others. The number of teams observed is relatively small, 
however, and further work is required to exclude the possibility 
that this finding is simply due to personality and leadership 
differences within the particular surgical teams observed. 

This study has a number of limitations. It would have been 
desirable to rate series of operations using both the original Oxford 
NOTECHS and the new scale in parallel, to evaluate how well we 
had achieved improved scalability whilst maintaining concurrent 
validity. This proved impossible as our observers considered they 
would be unable to switch between the scales and remain 
objective, whilst providing additional NOTECHS-trained observ- 
ers was impracticable. As noted above, our findings about average 
Oxford NOTECHS II scores for different types of surgery require 
confirmation, as does our conclusion that a single observer may be 
able to score NOTECHS II reliably, whilst the problem of scoring 
anaesthetic behaviour adequately within a whole-team scale 
requires further work. Neither Oxford NOTECHS II nor other 
attempts to score surgical team behaviour has yet demonstrated a 
clear ability to predict clinical outcomes: our findings suggest that 
a combination of Oxford NOTECHS II and glitch count is more 
likely to be able to achieve this than either scale alone, but 
numerous other factors, particularly around post-operative care, 
are likely to confound this analysis. Our aim here is the 
exploration and development of a usable scale, which is an 
important and necessary step towards this goal. 

Conclusion 

In conclusion we have found that the new Oxford NOTECHS 
II has retained good levels of inter-rater reliability making it 
potentially reliable with a solo observer. Vahdation studies showed 
the expected association with WHO time-out performance, and an 
orthogonal relationship with glitch rate which is theoretically 
explicable and may be valuable in permitting the use of 
complementary tools for overall assessment of teams. It offers a 
higher degree of precision than the previous version of the 
approach, potentially providing greater sensitivity in measure- 
ment. We have also found it to provide a relevant insight into the 
variation of theatre team's non-technical skills performance across 
surgical disciplines and hospitals. The approach appears to be a 
useful tool for exploring surgical performance for future research 
studies, but also may have apphcations as a quality improvement 
tool in its own right. 
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Figure 3. Distribution of NOTECHS II scores for the whole team and the three sub-teams (Anaesthetists, Nurses, Surgeons). 
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